String Metrics and Word Similarity applied to Information Retrieval

Over the past three decades, Information Retrieval (IR) has been studied extensively. The purpose of information retrieval is to assist users in locating information they are looking for. Information retrieval is currently being applied in a variety of application domains from database systems to web information search engines. The main idea of it is to locate documents that contain terms the users specify in their queries. The thesis presents several string metrics, such as edit distance, Q-gram, cosine similarity and dice coefficient. All these string metrics are based on plain lexicographic term matching and could be applied to classical information retrieval models such as vector space, probabilistic, boolean and so on. Experiment results of string distance metrics on real data are provided and analyzed. Word similarity or semantic similarity relates to computing the similarity between concepts or senses of words, which are not lexicographically similar. WordNet, can be classified into two categories: one uses solely semantic links, the other combines corpus statistics with taxonomic distance. Five similarity measures belonging to these two categories are selected to conduct the experiment on the purpose of comparison. Hierarchical clustering algorithms including both single-linkage clustering and complete-linkage clustering are studied by employing word similarity measures as clustering criteria. Stopping criteria including Calinski & Harabasz, Hartigan and WB-index are used to find the proper hierarchical level in the clustering algorithms. Experiments on both synthetic datasets and real datasets are conducted and the results are analyzed. Acknowledgements I would like to thank Dr. Pasi Fränti for the advice, encouragement and support he provided to me in supervising this thesis eort. I would also like to thank Qinpei Zhao, for her critical analyses, technical advice and recommendations. Special thanks go to Dr. Olli Virmajoki for his reviews and recommendations.